Mathematics: finding rank of matrix